


PERCEPTION 

VISUAL, AUDITORY, MULTIMODAL 




Harder than it seems.. . 
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Perceptual Systems 


























Perceptual systems are... 

- a snapshot of the world? 

- an interpretation of theS^Jjp? 
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taste & 
smell 


touch 


audition 
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What are the goals of vision? 

-Accurately record the color, intensity, 
etc. of light? 

-NO! Recognize objects... 

• What objects are out there? 

• Where are they? 

• How are they moving? 
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Harder than it "looks 




ambiguity! 

= more than one possible interpretation 


"Whatever job you give him, I have no doubt that 
he will be fired with enthusiasm." 
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Harder than it "looks 




Have to construct a 3-D representation from 
a 2-D pattern of light on the retina; this is an 
"indeterminant" problem. 




From Front of the Eye to Back of the Brain 


Axons Bipolar S3 ^ 

s-JL* - 

Cdl Fovea ^ 


Light-► t 



Lateral Gencu late 
Nucleus (LGH) 


Primary 
Visual 
Cortex (VI) 


Optic 

Nerve 


Inverse optics problem 


■ 




Consider that objects 


- can be seen from many orientations 
-can be lit in different ways 

- can be seen at different distances 

- can be occluded by other objects 
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Example: Determining color, 



Think about how to design a system 
that detects whether an object is red, 
blue, or purple... 





The perceived color of objects depends on what frequencies 
of light those objects absorb versus reflect. g 



• Assume red objects reflect 80% of the red light 
that hits them and only 10% of the blue light. 

• Assume purple objects reflect 50% red, 50% blue. 

• Assume blue objects reflect 80% of the blue light 
and 10% of the red. 
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Simple color determination 


■ 



Light comes in with 100 units of red and 
100 units of blue: 


Red object Purple object Blue object 

Red light 80 50 10 

Blue light 10 50 80 


11 




Simple color determination 





• But what if the light source changes? 

- is perception the same? 



fri, 

.com 
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dreom/bime 
















Light comes in with 10 units of red and 
190 units of blue: 



Red object 

Purple object 

Blue object 

Red light 

8 

5 

1 

Blue light 

19 

95 

152 
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Not so simple color 
termination 




■ 



Light comes in with 100 units of red and 
100 units of blue: 



Red object 

Purple object 

Blue object 

Red light 

80 

50 

10 

Blue light 

10 

50 

80 



Light comes in with 10 units of red and 
190 units of blue: 


Red object Purple object Blue object 

Red light 8 5 1 

Blue light 19 95 152 14 
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The brain solves this by normalizing 
comparing colors across the scene] 



Blue square second 
from right 


Yellow square second 
from left 
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How does the brain deal wit 
biguity? 





Three key strategies 

Use context (contrasts) 

Make assumptions 

Use "top-down" information 
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How does the brain deal wit 
biguity? 





Three key strategies 

• Use context (contrasts) 

• Make assumptions 

• Use "top-down" information 
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Color constancy 






















The visual system compares the input it gets 
from region to region. 


Sharp differences tend to get accentuated. 
This is good for processes like finding edges. 

Constant factors tend to get ignored. 

This is good for dealing with variability. 
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Three key strategies 


• Use context (contrasts) 

• Make assumptions 

• Use "top-down" information 
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• Light source comes from above. 

• For fun, also take a look at: 

- www.michaelbach.de/ot/fcs_hollow-face/index.html 
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Another example 








Another example 


■ 






Another example 














Assumptions about groupin 




some Gestalt principles 


Proximity 
Similarity 
Good continuation 
Closure ("Pragnanz") 
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• • • • 

• • • • 

t • • • 

• • • • 

Do you see rows or columns? 

Could be either, but objects are closer together 
in the rows, so we tend to see rows. 
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O • O • 0 

O • O • 0 


O • 0 • 0 

O • O • 0 


Do you see rows or columns? 

Tend to group together similar items, so in 
this case people tend to see columns. 
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Four lines? Two "v" shapes? Two long lines? 


Tend to assume that entities that lie along 
a continuous line or surface are "one" object. 
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What do you see? 


We tend to assume that shapes are closed and 
to "fill in" information that appears occluded. 
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Recap: assumptions 


■ 





• The visual system makes assumptions about 
typical viewing conditions and about what 
parts of a visual scene go together. 

- not conscious or strategic (can't be turned off) 

- are sometimes misleading, but usually work well 
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Three key strategies 


• Use context (contrasts) 

• Make assumptions 

• Use "top-down" information 
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Knowledge and object recognition 




Recognition is faster for objects in a 
coherent scene than in a jumbled one. 







Knowledge and object recognition 



What does this say about our model of visual 
erception? 





Visual 

Processing 


Input 
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Vision: Summary 



Vision is a process that creates a 
3d representation from 2d input 

- perceiving the input 

- integrating with context 

• memory, the world 

- interpreting ambiguity 
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• http://illusionofthevear.com/ 

• http://gethighnow.com/ (also some auditory ones) 
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touch 


taste & 
smell 


vision 
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AUDITORY PROCESSING 


• Does the mind "hear" the world for 
what it is? 

- or instead offer an interpretation of it? 
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Speech spectrogram 


■ 




• The speech stream is continuous. 

oeoo 

Hz 
3000 

2400 

1SOO 

1200 

GOO 

-Li 

"four score and seven years ago" 



Hey, where are the words?? 
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Where are the word boundaries? 

"Olive, the other reindeer..." 

(Rudolph the Red-nosed Reindeer) 

"There's a bathroom on the right..." 

("Bad Moon Rising" Creedence Clearwater Revival) 

” Mondegreens" 
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How a particular sound (phoneme) 
is pronounced depends on: 

- the speaker 

- the speech rate 

- emotional factors 

- the sound environment ("coarticulation") 


56 







1 



■ 


• The auditory system "exaggerates" 
differences between sounds. 

- For example, sounds "in between" BA and PA are 
heard as one or the other. 

• Babies start out showing categorical 
perception for all possible sound contrasts. 

• If those contrasts don't matter in your 
language(s), the auditory system learns to 
ignore the difference. 
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• Like the visual system, the auditory system 
uses context: 

• topic of conversation 

• recent words 

• recent sounds 

• Context can also come from other sensory 
modalities ... 
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Audition: Summary 


There is also a lot of ambiguity in 
auditory processing (especially 
speech) 

Built-in strategies for dealing with 
variability (categorical perception) 
help. 

Also make use of context 
information and cues from other 
senses. 
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Perceptual Systems 



touch 


audition 


vision 
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• Context guides taste perception 






















Taste & Smell 


Context guides taste perception 
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Summary: Perception 



Perception involves 

- integration across senses 

- use of contextual knowledge 

• from the world 

• from memory 

- interpretation of input 
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Summary: Perception 




Perceptual systems are./: — m 

- a snapshot of the wor l d? 

- an interpretation of the world 
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Thought question 


■ 



Imagine you are building an artificial vision system. 
Your system first uses grouping principles to 
determine how many objects are in the scene. 

Next, the features of each object are extracted. 
Finally, each object is identified serially, by 
comparing the features for that object with feature 
lists stored in memory. 

How will your system behave if you ask it to identify 
an object presented in isolation as compared with an 
object in a complex scene? 

Is this similar to the pattern we would expect to see 
for humans? Why or why not? 
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